Overview of QA4MRE Main Task at CLEF 2013

نویسندگان

  • Richard F. E. Sutcliffe
  • Anselmo Peñas
  • Eduard H. Hovy
  • Pamela Forner
  • Álvaro Rodrigo
  • Corina Forascu
  • Yassine Benajiba
  • Petya Osenova
چکیده

This paper describes the Question Answering for Machine Reading (QA4MRE) Main Task at the 2013 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Machine Reading on Biomedical Texts about Alzheimer's disease, and Japanese Entrance Exams. This paper describes the preparation of the data sets, the definition of the background collections, the metric used for the evaluation of the systems’ submissions, and the results. We introduced two novelties this year: auxiliary questions to evaluate systems level of inference, and a portion of questions where none of the options were correct. Nineteen groups participated in the task submitting a total of 77 runs in five languages.

برای دانلود رایگان متن کامل این مقاله و بیش از 32 میلیون مقاله دیگر ابتدا ثبت نام کنید

ثبت نام

اگر عضو سایت هستید لطفا وارد حساب کاربری خود شوید

منابع مشابه

NAIST at the CLEF 2013 QA4MRE Pilot Task

This paper describes the Nara Institute of Science and Technology’s system for the entrance exam pilot task of CLEF 2013 QA4MRE. The core of the system is a similar to the system for the main task of CLEF 2013 QA4MRE. We use minimum error rate training (MERT) to train the weights of the model and also propose a novel method for MERT with the addition of a threshold that defines the certainty wi...

متن کامل

Overview of QA4MRE 2013 Entrance Exams Task

This paper describes the Question Answering for Machine Reading (QA4MRE) Entrance Exams at the 2013 Cross Language Evaluation Forum. The data set of this task is extracted from actual university entrance examinations as-is, and therefore includes a variety of topics in daily life. Another unique feature of the Entrance Exams task is that questions are designed originally for testing human exami...

متن کامل

Inter-Sentence Features and Thresholded Minimum Error Rate Training: NAIST at CLEF 2013 QA4MRE

This paper describes the Nara Institute of Science and Technology’s system for the main task of CLEF 2013 QA4MRE. The core of the system is a log linear scoring model that couples both intra and intersentence features. Each of the features receives an input of a candidate answer, question, and document, and uses these to assign a score according to some criterion. We use minimum error rate trai...

متن کامل

Overview of QA4MRE at CLEF 2011: Question Answering for Machine Reading Evaluation

This paper describes the Question Answering for Machine Reading (QA4MRE) task at the 2012 Cross Language Evaluation Forum. In the main task, systems answered multiple-choice questions on documents concerned with four different topics. There were also two pilot tasks, Processing Modality and Negation for Machine Reading, and Machine Reading on Biomedical Texts about Alzheimer's disease. This pap...

متن کامل

QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation

This paper describes the methodology for testing the performance of Machine Reading systems through Question Answering and Reading Comprehension Tests. This was the attempt of the QA4MRE challenge which was run as a Lab at CLEF 2011–2013. The traditional QA task was replaced by a new Machine Reading task, whose intention was to ask questions that required a deep knowledge of individual short te...

متن کامل

ذخیره در منابع من


  با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید

عنوان ژورنال:

دوره   شماره 

صفحات  -

تاریخ انتشار 2013